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ABSTRACT 

In this study, we discuss the use of generative evaluation as an innovative approach in the analysis of 
qualitative interview data for evaluating simultaneous renewal of Iowa State University’s PT3 grant. 
Goodlad’s (1994) simultaneous renewal framework forms the basis of our argument. We focused on the 
CREATER+ model designed to explain the complexities of understanding simultaneous renewal in this 
multifaceted university/K-12 partnership. 

OVERVIEW: GOALS 

TechCo (Technology Collaborators for Simultaneous Renewal), Iowa State University’s (ISU) Preparing 
Tomorrow's Teachers to Use Technology (PT3) implementation grant project is aimed at developing systemic 
change in K-12 schools and teacher education programs through simultaneous renewal. A working definition of 
educational renewal is an ongoing process of self-examination, reflection, and change (Rafferty, 2003). In this 
regard the project is focused on renewing teacher education programs through the extensive and effective use 
and integration of technology in student-centered learning environments. John Goodlad’s (1994) theory of 
simultaneous renewal and constructivist theory in learning and teaching provide the two major frameworks of 
TechCo. 

In order to accomplish project goals, TechCo works collaboratively with many partners: university partners 
including Curriculum and Instruction Department, Center for Technology in Learning and Teaching (CTLT) and 
the College of Engineering; three elementary schools from the Des Moines area including students, teachers, 
principals, pre-service, and in-service teachers, as well as technology coordinators together form the K-12 
partners; The Area Education Agency (AEA) an outside educational organization partner and Apple computer as 
the business partner. Together, these partners form an interconnected consortium needed for simultaneous 
renewal. In order to accomplish simultaneous renewal in these organizations the research team focused on 
gathering and analyzing data by engaging all partners in the evaluation process. 

“Engagement of all Stakeholders in the Evaluation Process: Generative Evaluation ” 

The progress of the TechCo project has been evaluated formatively based on the surveys for both pre-service and 
in-service teachers during the first two years of the project. Two major surveys, Survey of the Use and 
Integration of Computer-Related Technology and Cultural Diversity Awareness Inventory have been conducted 
for teachers in this regard and at the end of the 3-year period the same surveys were used to collect additional 
data. 

The three years of extensive studies in both K-12 schools and at ISU called for an innovative and complex 
approach for conducting TechCo evaluation. To do this, the evaluation team adopted several techniques from 
many models including the CREATER Model of Havelock and Zlotolow (1995), the Vision 2020 Model or 
WorkSpan/LINC Model, and European experiences in the renewal of teacher education programs in the use of 
technology. Some elements of the Concerns Based Adoption Model by (Hall, George and Rutleford, 1979) were 
also used. The final framework derived from this eclectic approach and the application of change models to the 
process of evaluation resulted in the ‘generative evaluation’ (Davis & Kemis, 2002). Using CREATER and 
other models in-depth structured interviews were conducted with all leaders. 
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These leaders included Dean of the COE, project leaders in the COE, principals and technology coordinators in 
the elementary schools. Other qualitative data resources comprising (student artifacts, written documents such 
as syllabi, yearly reports, videotapes, pictures, etc.) were also utilized as a result of this new approach. 

The evaluation team called this new approach " Generative Evaluation" since the data gathered from the field 
showed its’ value for the change in the institutions as an important byproduct. This innovative generative 
evaluation technique is a key overarching strategy within TechCo and the partnerships and inter-project 
collaboration. There is merit in using this approach to evaluation because it can “guide leaders and change 
agents on their pathway to the future". It was created to answer the need to develop an appropriate methodology 
that could be applied to inform a holistic perspective in complex systems (Davis, Kemis, & Johnson, 2003, p.2). 

METHODS 

In the analysis of structured interview data of TechCo, it was mainly to be utilized from two data analysis 
techniques: analytic induction and hermeneutical analysis. 

Analytic Induction (interpreting based on change models) 

This technique is based on the examination of facts, situations, phenomena and events of the study and then 
develops the hypothesis, which explains these conditions. 

If the initial hypotheses do not match with the field studied, then it must be revised. This process continues until 
reaching the full explanation of situations. Ratcliff (2000b) describes this point saying that analytic induction is 
“inductive, rather than deductive, reasoning is involved, allowing for modification of concepts and relationships 
between concepts which occur throughout the process of doing research, with the goal of most accurately 
representing the reality of the situation.” And so this methodology refers to several steps in data analysis in 
order to “move from raw data to proposed answers and solutions” (Willis et al., 2003). 

TechCo’s evaluation team is using a wide range of data collection sources such as interviews (both structured 
and focus groups), surveys, document analysis, etc., to collect rich data leading to triangulation of data sources 
(Guba & Lincoln, 1995). This has been evident especially in the collection of interview data, which provides the 
primary data for ‘generative evaluation'. Havelock and Zlotolow (1995) original CREATER model was found 
to be limited in terms of explaining the ‘renewal’ phase of systemic change, because it yielded no information 
about the processes and interactions during the renewal process. Therefore, Bosserman’s (1998) model of 
institutional change was adopted in order to bridge this gap with the formal structures of organizations. 
Bosserman’s model of institutional change was utilized to understand the interviewee’s general understanding of 
the changes required for renewal in the specific organization. Researchers using Bosserman’s model of 
institutional change are encouraged to think in terms of formal (wider environment) and informal structures 
(immediate environments) in which people operate and interact (Davis, Kemis, & Johnson, 2003). Other models 
are also used where appropriate (See Ellsworth for a survey of change models, 2002). 

These three models have been applied to create a unique approach for the study and the project team called it 
‘generative analysis' (Thompson, Schmidt and Davis, 2003). In the data analysis process, this approach can be 
considered as the main hypothesis of the project by shedding light on the understanding of systemic change. 
After transcribing the interview protocols, these transcribed data have been coded around the elements of the 
CREATER+ model. Even though there is an extensive amount of data gained during the three-year period, it has 
been reduced and summarized using the seven elements of the CREATER+ model in order to get a clearer 
understanding of the change process and simultaneous renewal within TechCo. Based on the volume and type of 
data achieved through this process a hermeneutical analysis was deemed appropriate for shedding light on the 
context of this evaluation 

Hermeneutical Analysis (understanding in the context) 

By nature, hermeneutical analysis is an inteipretive approach as it emphasizes the importance of the views of 
participants based on their experiences and their standpoint. One implication of this methodology is the idea of 
‘hermeneutic circles’, which is explained by Klein and Meyers (1998) cited in Willis et al., (2003). 

The idea of hermeneutic circles suggests that we come to understand a complex whole from 
preconceptions about the meanings of its’ parts and their interrelationships... the movement of 
understanding is constantly from the whole to the part and back to the whole. Our task is to extend in 
concentric circles in unity of the understood meaning. 
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The TechCo evaluation team adopted a more interpretivist approach using hermeneutical analysis because the 
data contains information from one university and four different schools, which demand local explanations of 
facts and events. For instance, the four schools studied have very different characteristics in terms of 
administrative support, teacher attitudes, and priorities in the use of technology, etc. The initial examination of 
the interview data shows this very clearly. Within TechCo' s project, the situated understanding is important as 
well as understanding multiple perspectives of various stakeholders. Thus, using hermeneutical techniques when 
describing different school settings, different school experiences as well as the different stories of K-12 people 
and ISU faculty (Technology scholars (faculty); cohort student experiences; student artifacts; etc.) was a holistic 
approach to this phenomenon understudy. 

RESULTS 

Each structured interview with project leaders has been analyzed using CREATER+ model and aforementioned 
analysis techniques. One sample interview chart is shown in Figure 1. 

After completing the cross-case analysis and member check of the interview results the TechCo research team 
will write up the results. Since this is an ongoing process, this paper will not focus on overall results of the 
project. However, preliminary results show that the collaboration between K-12 schools and Iowa State 
University's teacher education program, through TechCo is renewing educational environments with its 
emphasis on technology integration and learner centeredness. This is congruent with the tenets of simultaneous 
renewal. 

Simultaneous renewal in general is both multifaceted and fragile requiring sustained, visionary and well- 
informed leadership (Sherry, 2003). “The development of ‘generative evaluation’ for simultaneous renewal... in 
teacher education holds promise for the application of this robust approach that is transferable and applicable to 
other projects and partnerships” (Davis, Kemis, & Johnson, 2003, p. 16). In this regard, ISU shares its 
experience and expertise with a very successful faculty development model, human resources such as faculty, 
staff, and graduate and undergraduate students whereas K-12 schools are involved fully in these efforts with all 
administrators, teachers, tech support staff and students. 



Sample Interview Analysis (12/17/2001) 




Figure 1. Sample of Interview Analysis (Without the detail of the central circle) 

Significance/ Impact 

Goodlad’s theory of simultaneous renewal is working in practice in ISU’s PT3 TechCo initiative, providing rich 
cases for theoreticians and practitioners. As observed by Goodlad: 
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What comes first, good schools or good teacher education programs? The answer is that both must 
come together. There are not now the thousands of good schools needed for the internships of tens of 
thousand of future teachers. The long-term solution — unfortunately, there is no quick one — is to renew 
the two together. There must be a continuous process of educational renewal in which colleges and 
universities, the traditional producers of teachers, join schools, the recipients of the products, as equal 
partners in the simultaneous renewal of school and the education of educators (Goodlad, 1994, p. 2). 

This project is one of the best examples of integration of technology rich environments into learning and 
teaching through university/K-12 partnerships. TechCo’s team keep disseminate further outcomes and products 
of these experiences after the analysis and interpretation of the project’s data is completed. 
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